Building a Pet Store Terraform Provider

Even though the Terraform provider registry has almost every provider we can think of, there is a chance that a provider we need does not yet exist. Perhaps we want to use Terraform to interact with resources of a proprietary API internal to our company. If we want to manage resources that don't yet exist in the Terraform provider ecosystem, we will need to write a provider for that API. The good news is that writing a Terraform provider is relatively simple. The thoughtful folks at HashiCorp provide great documentation, SDKs, and tools to make building a provider a breeze.

Resources for building custom providers#

HashiCorp provides an extensive set of tutorials for building custom providers. We highly recommend reviewing the content if you intend to build your own custom provider.

We will not cover all of the code, but we will dive into the most interesting parts. We've done my best to keep to only the most simple implementation; however, simple is not always elegant.

Additionally, our pet store custom provider uses the Terraform plugin SDK v2 rather than the new (at the time of writing) Terraform plugin framework. We chose this path as the majority of existing providers use the SDK v2, and the Terraform plugin framework has not reached stability yet. If you are interested in weighing the benefits, read the “Which SDK Should I Use?” article from HashiCorp.

Now that we have established a foundation of content and learning, let's proceed to the code.

The pet store provider#

Our pet store Terraform provider is just another Go application. Most of the interactions between Terraform and the provider are handled at the Terraform SDK level, and very little gets in the way of the provider developer. Let's start off by taking a look at the directory structure of the provider:

It's a standard Go application with an entry point in main.go. Let's start at the top and work our way down the files. The first on the list is the Makefile:

The preceding Makefile offers some helpful build tasks and environmental configuration. For example, make or make install will build the provider for the current architecture and place it in the ~/.terraform.d/plugins directory tree, which will enable us to use the provider locally without publishing it to the registry.

Next, we have the docker-compose.yml file. Let's take a look:

The docker-compose.yml file runs the pet store service and exposes the gRPC service on port 6742. The pet store service stores pets in an in-memory store, so to wipe out the pets currently stored, just restart the service. We'll talk more about starting and stopping the service later in the lesson.

Next up, we have examples/main.tf. Let's see what an example of defining our pet resources will look like:

In the preceding main.tf file, we can see the provider registered and configured to use the local pet store service. We can also see the definition for two petstore_pet resources, Thor and Tron. After the resources, we define a petstore_pet data source. We will walk through bits of this file in more detail later in the lesson.

The main reason we like to see main.tf before we get into the code is that it will give us an idea of the interface we want to achieve in the provider implementation. We believe seeing the usage of the provider will help us to better understand the provider implementation.

The rest of the source code is all in Go, so rather than going from top to bottom, we're going to move to the entry point in main.go and dive into the actual implementation:

Well, main.go is simple enough. All we are doing in main is starting a plugin server via the Terraform plugin SDK v2 and providing it with an implementation of our pet store provider. Let's next look at the petstore.Provider implementation in internal/provider.go:

There are only two functions in provider.go. The Provider function creates an *schema.Provider that describes the schema for configuring the provider, the resources of the provider, the data sources of the provider, and the configure function for initializing the provider. The resource map for the provider contains resources by a string name and their schemas. The schemas for each of the structures describe the domain-specific language to Terraform for interacting with their fields and resource hierarchies. We will examine the schemas for these structures in more detail soon.

Next, let's look at the configure function in provider.go:

The configure function is responsible for handling provider configuration. Note how the host data described in the preceding Provider schema is available via the data argument. This is a common pattern we will see throughout the provider. We use the host configuration data to construct the client for the pet store service. If we are unable to construct a pet store client, we append a diag.Diagnostic structure to the slice of diag.Diagnostics. These diagnostic structures inform Terraform of an event of varying severity occurring in the provider. In this case, it is an error if we are unable to build the client, which should be communicated back to the user. If all goes well, we return the client instance and an empty slice of diag.Diagnostics.

Next, let's examine the pet store data source.

Implementing the pet store data source#

The pet store data source is a bit simpler than the resource implementation, given that a data source is intended as a way for Terraform to pull in data from an external API and is read-only in this case. The pet store data source is defined in internal/data_ source_pet.go.

There are three functions of primary interest in the pet store data source. We will approach them one at a time. Let's start with the dataSourcePet function:

The preceding function creates the *schema.Resource data source by providing a schema for the data being provided via getPetDataSchema.ReadContext expects a function that is responsible for translating the input schema, querying the external API, and returning data to Terraform that matches the structure defined in the schema.

The definition of getPetDataSchema is located in internal/schema.go, and it is helpful to review it prior to examining the code in dataSourcePetRead. We will break down the function into two parts, the input and the computed output:

The preceding schema describes the data structure for the pet store pet data source. Each of the top-level keys is marked as optional and will be used to filter the data source. For example, the name key specifies that it is optional, is of type string, and should be validated with the validateName function. We will examine validations in more detail later in the section.

The following is the schema for the output of the data source:

The pets key contains all the Computed values, which means each of the values is read-only. These represent the list result of the query.

Now that we have a better understanding of the data schema we are working with let's continue with the implementation of dataSourcePetRead:

In dataSourcePetRead, we instantiate a client for the pet store service, populate the filter criteria from the data schema supplied, and then set the pets key in the data argument with the pets returned from the pet store service in the key value format specified by the schema. The flattenPets function is responsible for transforming the protobuf structures we receive from the pet store service into the format expected by the schema. If we are interested in the implementation, it is not terribly elegant, but it is simple.

We purposely didn't mention the data.SetId function. We are setting the value of that to a value that will cause the data to be fetched from the pet store service each time. Terraform identifies that data has changed if the ID for that data has changed. This ensures that the ID changes each time the function is executed.

In the configure function, we created the pet store client, so how did we gain access to that client in the data source? We can find the answer to that in the clientFromMeta function:

The clientFromMeta function takes the meta interface{} argument passed into the ReadContext function and casts it as the pet store client. The meta variable contains the variable returned in the configure function. This is not as intuitive as we would like, but it is effective.

With the code described previously and some helpers from internal/data_source_ pet.go, we have implemented a filtered data source to the pet store API that we can use in Terraform configuration files. Next, let's take a look at how we handle CRUD interactions for pet resources.

Implementing the Pet resource#

The implementation for the Pet resource follows many of the same patterns as the pet store data source, but with the pet resources, we also need to implement create, update, and delete interactions in addition to read. Unless otherwise stated, the code we cover for the pet resource implementation is in internal/resource_pet.go.

Let's start by examining the resourcePet function, which is the function called when we created the provider schema:

Just like the pet store data source, the pet resource defines handlers for each CRUD operation as well as a schema. Before we get into the CRUD operations, let's first look at the schema, which is in internal/schema.go:

The schema defined here is simpler than the data source schema since we are not defining query filters. Note that the id key is computed, but all the others are not. The id value is generated by the pet store service and is not to be specified by the user.

Since these values are specified by the user as a string, validation becomes more significant. For a better user experience, we want to provide feedback to a user when a value is invalid. Let's take a look at how we validate the type field with the validateType function:

The validateType function returns a validation constructed with each valid value of the enumeration. This prevents a user from entering a string value for a pet type that is not supported in the pet store. The rest of the validations take a similar approach to validating the range of input values.

Now that we have explored the schema, we are prepared to explore the CRUD operations. Let's start with the read operation:

The resourcePetRead function fetches the pet store client from the meta argument and then finds the pet by ID in the store. If the pet is found, the data argument is updated with data from the pet.

That's simple enough. Next, let's look at create:

The resourcePetCreate function follows a similar pattern. The difference is that the pet is constructed from fields in the data argument, and then the pet store API is called to add the pet to the store. In the end, the ID for the new pet is set.

Next, let's look at update:

The resourcePetUpdate function combines parts of read and create. Initially, we need to check to see whether the pet is in the store and fetch the pet data. If we don't find the pet, we return an error. If we do find the pet, we update the fields of the pet and call UpdatePets on the pet store client.

The delete operation is relatively trivial, so we will not dive into it here. If we want, we can take a look at resourcePetDelete to see for ourself.

At this point, we have now implemented the pet resource and are ready to see our Terraform provider in action.

Running the pet store provider#

Let's run our Terraform provider:

Now that we have a fully implemented pet store provider, the fun part is running it. From the root of the pet store provider, run the following commands:

The preceding commands will start the pet store service using docker-compose, build and install the provider, move it into the example directory, and finally, use init and apply to create our desired state containing our pets.

When init executes, we should see something like the following:

Yay! The provider is installed, and Terraform is ready to apply our resources.

After Terraform has applied the resources, we should see the following output:

We can see from the preceding output that both of our resources, Tron and Thor, have been added, and our data source when queried with no filters returned each of the pets. Lastly, we can see the thor output was returned, containing the data for Thor.

Let's review examples/main.tf again and see where the thor output came from:

In the preceding main.tf file, we defined a pet_name variable with the value of Thor. We then queried the pet store data source, providing no filters but depending on the completion of both of the resources in the file. Lastly, we output a key of thor, with the value being a query that matches only when pet.name equals var.pet_name. This filtered the data source for only pets named Thor.

We can now use any of the Terraform skills we've learned thus far to manipulate pet store resources. There really wasn't all that much code to implement.

Publishing custom providers#

Anyone can publish a provider to the Terraform Registry by logging into it using a GitHub account. Again, HashiCorp has excellent documentation on how to publish a provider. We will not walk through the process in this course, as the documentation for “Release and Publish a Provider to the Terraform Registry” is likely sufficient if we have reached this far in our journey of building our own Terraform provider.